Ingester memory improvements by adjusting prealloc #4344
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Joe Elliott <[email protected]>
Signed-off-by: Joe Elliott <[email protected]>
joe-elliott
requested review from
mdisibio,
mapno,
yvrhdn,
zalegrala,
electron0zero,
ie-pham,
stoewer and
javiermolinar
as code owners
Signed-off-by: Joe Elliott <[email protected]>
Signed-off-by: Joe Elliott <[email protected]>
Signed-off-by: Joe Elliott <[email protected]>
Signed-off-by: Joe Elliott <[email protected]>
zalegrala
approved these changes
Nov 19, 2024
| return p.make(sz) | ||
|
|
||
| // Find the right bucket. | ||
| bkt := sz / p.bktSize |
There was a problem hiding this comment.
I suppose its a trad-off here to alloc or do the math twice.
There was a problem hiding this comment.
this doesn't really "alloc" in any way we are concerned about. we're always conscious of allocating memory that escapes to the heap. everything else is cheap. whether this is stored on the stack of just stays in a register it would likely be undetectable from a tempo perf standpoint.
|
|
||
| b := p.buckets[bkt].Get() | ||
| if b == nil { | ||
| sz := (bkt + 1) * p.bktSize |
There was a problem hiding this comment.
Does this alloc, or is the compiler smart enough to see its only used once?
There was a problem hiding this comment.
you're really testing my knowledge of compilers :). my guess is this value would never leave a register
There was a problem hiding this comment.
You can do escape analysis in Go with go build -gcflags "-m" <file>. It does not escape :P
|
This PR must be merged before a backport PR will be created. |
1 similar comment
|
This PR must be merged before a backport PR will be created. |
github-actions bot
pushed a commit
that referenced
this pull request
Nov 19, 2024
* remove trace ids Signed-off-by: Joe Elliott <[email protected]> * linear buckets Signed-off-by: Joe Elliott <[email protected]> * changelog Signed-off-by: Joe Elliott <[email protected]> * tuney tune Signed-off-by: Joe Elliott <[email protected]> * metric misses and increase pool size Signed-off-by: Joe Elliott <[email protected]> * lint Signed-off-by: Joe Elliott <[email protected]> --------- Signed-off-by: Joe Elliott <[email protected]> (cherry picked from commit f71c4c6)
3 tasks
joe-elliott
added a commit
that referenced
this pull request
Nov 19, 2024
* remove trace ids Signed-off-by: Joe Elliott <[email protected]> * linear buckets Signed-off-by: Joe Elliott <[email protected]> * changelog Signed-off-by: Joe Elliott <[email protected]> * tuney tune Signed-off-by: Joe Elliott <[email protected]> * metric misses and increase pool size Signed-off-by: Joe Elliott <[email protected]> * lint Signed-off-by: Joe Elliott <[email protected]> --------- Signed-off-by: Joe Elliott <[email protected]> (cherry picked from commit f71c4c6) Co-authored-by: Joe Elliott <[email protected]>
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-4344-to-r174 origin/r174
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x f71c4c6ccbd81b75826b434d6bf2b82174a757c5When the conflicts are resolved, stage and commit the changes: If you have the GitHub CLI installed: # Push the branch to GitHub:
git push --set-upstream origin backport-4344-to-r174
# Create the PR body template
PR_BODY=$(gh pr view 4344 --json body --template 'Backport f71c4c6ccbd81b75826b434d6bf2b82174a757c5 from #4344{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Create the PR on GitHub
echo "${PR_BODY}" | gh pr create --title '[r174] Ingester memory improvements by adjusting prealloc' --body-file - --label 'type/bug' --label 'backport' --base r174 --milestone r174 --webOr, if you don't have the GitHub CLI installed (we recommend you install it!): # Push the branch to GitHub:
git push --set-upstream origin backport-4344-to-r174
# Create a pull request where the `base` branch is `r174` and the `compare`/`head` branch is `backport-4344-to-r174`.
# Remove the local backport branch
git switch main
git branch -D backport-4344-to-r174 |
joe-elliott
added a commit
that referenced
this pull request
Nov 20, 2024
* remove trace ids Signed-off-by: Joe Elliott <[email protected]> * linear buckets Signed-off-by: Joe Elliott <[email protected]> * changelog Signed-off-by: Joe Elliott <[email protected]> * tuney tune Signed-off-by: Joe Elliott <[email protected]> * metric misses and increase pool size Signed-off-by: Joe Elliott <[email protected]> * lint Signed-off-by: Joe Elliott <[email protected]> --------- Signed-off-by: Joe Elliott <[email protected]> (cherry picked from commit f71c4c6)
3 tasks
joe-elliott
added a commit
that referenced
this pull request
Nov 20, 2024
* remove trace ids Signed-off-by: Joe Elliott <[email protected]> * linear buckets Signed-off-by: Joe Elliott <[email protected]> * changelog Signed-off-by: Joe Elliott <[email protected]> * tuney tune Signed-off-by: Joe Elliott <[email protected]> * metric misses and increase pool size Signed-off-by: Joe Elliott <[email protected]> * lint Signed-off-by: Joe Elliott <[email protected]> --------- Signed-off-by: Joe Elliott <[email protected]> (cherry picked from commit f71c4c6)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Adjusts the use of prealloc for some nice memory improvements. Two adjustments were made:
Seeing ~30% working set reduction in ingesters.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]